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A Method and System for Super Resolution 

FIELD OF THE INVENTION 

The present invention relates to super resolution 
enhancement of sampled data. 

BACKGROUND OF THE INVENTION 



Super-resolution refers to the enhancement of 
resolutions for images of scenerv' or objects acquired by an image 
capture device, such as a scanner or a CCD video camera, beyond the 
resolution inherent in the capture device, and beyond other Limitations of 
15 the capture device. Super-resolution is used primarily to enhance the 

quality of captured images, and lo increase the information content 
therein. 

Many super-resolution techniques used with imaging 
sensors are based on micro scanning operations. When using micro- 

20 scanning operations, a scene or object being captured is sampled multiple 

times, each time with a sub pixel shift applied. 

Prior art techniques for super-resolution do not achieve 
exact super resolution, but rather they perform over-sampling. Exact 
super resolution is limited by the fill factor; i.e., the ratio between the 

25 area of the region sensitive to radiation and the pitch. The pitch is the 

area between centers of adjacent detectors in a sensor panel - 
equivalently, the pitch is the area between adjacent pixels. For example, 
if the sensitivity to radiation only extends over half the distance 
separating detectors, then the fill factor is 0.5 * 0.5 = 25%. With a 25% 

30 fill factor, exact super-resolution using prior art methods can be obtained 

up to a factor of two in resolution in each dimension, since the regions of 
sensi t ivity to mdtation are-enly-half the-sizc of the pitch, and can thus - bG- 
shifted by half-pixels in each dimension without overlapping. Additional 
super-resolution, beyond a factor of two, has been achieved by prior art 

35 methods at the expense of decreased contrast. 

For a fill factor of 80%, which is typical for image 
sensing devices, the limitation on exact super-resolution is 
y^jo^ = 1.12 . The experimental barrier is approximately at a factor of 

1.5, with some decrease in contrast. 
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5r art methods for obtaining super re^ution from 
captured images are based on estimating signal distonion. These 
methods perform optimal estimation of signal distortion from the 
capmred images, using Bayesian techniques and using criteria such as 
5 Maximimi Entropy. These methods are advantageous when image 

acquisition is made over a large distance (such as satellite data acquired 
from outer space), since atmospheric transmission and turbulence have a 
major impact on hmiting the resolution obtained. 

One such method is described in 1. Cheeseman, B. 
10 Kanefsky, R. Kraft, J. Stutz, R. Hanson, *'Super-Resolved Surface 

Reconstruction from Multiple Images,'' in Maximum Entropy and 
Bayesian Methods, G. R. Heidbreder (ed.), Kluwer, the Netherlands, 
1996, pgs. 293 - 308. This method is based on inverse graphics theory, 
and is used for ground modeling from outer space observations. An 
15 initial ground model is formed by letting each pixel "vote" on what the 

corresponding ground position should be, based upon the extent that the 
corresponding ground position contributes to that pixel. The initial 
ground model is then used to project what an image should be (i.e., 
predict each pixel value). The differences between the predicted pixel 
20 values and the observed pixel values are used to update the ground model 

until it cannot be further improved. This procedure produces an increase 
in both spatial resolution and gray-scale resolution. 

Another such method is described in A. Zomet and S. 
Peleg, "Applying Super-Resolution to Panoramic Mosaics," IEEE 
25 Workshop on Applications of Computer Vision, Princeton, Oct 1998. 

This method attains super resolution using an iterative method for 
mosaicing. Given a video sequence scanning a static scene, a panoramic 
image can be constructed whose field of view is the entire scene. Each 
region in the panorama is covered by many overlapping image segments, 
30 and this can be exploited to enhance its resolution. 

Another such method is described in Pearson, T. J. and 
Keadhead, A. C. S. "Image Formation by Self-Calibration in Radio 
Astronomy," Ann. Rev. Astron. Astrophys. 22, 1984, pgs. 97-130. This 
method-uses-non-linear-methQds,-and-is-based^n optimaLspectrum- 



35 estimation. 
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SUMMARY OF THE INVENTION 



The present specification concerns data obtained by 
sampUng a continuous signal, and describes methods and systems for 
5 enhancing the resolution of the data. For example, a barcode reader 

samples a barcode when scanning it, and the present invention can be 
used to enhance the resolution of the sampled data, thereby providing a 
better reconstruction of the barcode. For another example, a CCD 
camera samples an object or scene being viewed, to produce a digital 

10 image, and the present invention can be used to enhance the resolution of 

the digital image, thereby providing a better quality image. 

The present invention enhances sampled data by 
effectively decreasing the sampling period. When used with digital 
images, the present invention provides sub-pixel accuracy. An original 

15 image quantized into pixel area elements can be enhanced using the 

present invention to a finer granularity quantization with sub-pixel area 
elements. For example, the enhanced image can have one-eighth pixel 
granularity. 

The present invention overcomes the Limitation of prior 
20 an methods to achieve exact resolution improvement by more than a 

factor of 1.5. Using the present invention, any desired resolution 
improvement may be obtained by attaching an appropriately designed 
mask to a sensor plane of a capture device. 

There is provided in accordance with a preferred 
25 embodiment of the present invention a method for enhancing the 
resolution of an image sensing device, including the steps of attaching a 
mask to a panel of detectors in an image sensing device, generating 
multiple fields of view, the multiple fields of view being related to one 
another by sub-pixel shifts, acquiring multiple images with the image 
30 -sensing device from the multiple fields of view, and combining the 
multiple images into an enhanced image of higher pixel resolution than 
th e p Lxel resolutions o f the multiple images. 



There is further provided in accordance with a preferred 
embodiment of the present invention a system for enhancing the 



35 resolution of an image sensing device, including an image sensing device 

comprising a panel of detectors, a mask attached to said panel of 
detectors, a motion generator generating multiple fields of view, the 
multiple fields of view being related to one another by sub-pixel shifts, 
image acquisition circuitry housed within said image sensing device 

40 acquiring multiple images from the multiple fields of view, and a 



Atty. Docket No. 33,827 



-3- 



i^^^s the multiple imases into an enha^^ 



combiner comomng the multiple images into an enhaiWR image of 
higher pixel resolution than the pixel resolutions of the multiple images. 

There is still funher provided in accordance with a 
preferred embodiment of the present invention a method for enhancing 
5 the resolution of an image sensing device, including the steps of creating 

rephcas of fields of view using an optical element attached to an image 
sensing device, acquiring multiple images with the sensing device from 
the rephcas of fields of view, and combining the multiple images into an 
enhanced image of higher pixel resolution than the pixel resolutions of 
10 the multiple images. 

There is additionally., provided in accordance with a 
preferred embodiment of the present invention a system for enhancing 
the resolution of an image sensing device, including an image sensing 
device, an optical element attached to the image sensing device, the 
15 optical element being such as to create replicas of fields of view, image 

acquisition circuitry housed within the image sensing device acquiring 
multiple images from the rephcas of fields of view, and a combiner 
combining the multiple images into an enhanced image of higher pixel 
resolution than the pixel resolutions of the multiple images. 

20 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention wiU be more fully understood and 
appreciated from the following detailed description, taken in conjunction 
5 with the drawings in which: 

Figure 1 is an illustration of a one-dimensional cross- 
section of a prior art image sensing device, such as a CCD camera or a 
scanner; 

Figure 2 is a prior an illustration of a pixel sensitivity 
10 (apodization) function, for modeling averaged light intensity arriving at a 

detector; 

Figures 3A - 3C are illustrations of the results of 
computer simulations of a preferred embodiment of the present 
invention, as applied to CCD cameras; 
15 Figures 4A - 4D are further illustrations of the results of 

computer simulations of a preferred embodiment of the present 
invention, as applied to CCD cameras; 

Figures 5 A - 5C are illustrations of the results of 
computer simulations of a preferred embodiment of the present 
20 invention, as appUed to barcode readers; 

Figure 6 is an illustration of three samplings of an object 
by a sensing device, separated by sub-pixel shifts, in accordance with a 
preferred embodiment of the present invention; 

Figure 7 is a simplified illustration of a mask attached to 
25 a sensor panel of array detectors in an image sensing device, in 

accordance with a preferred embodiment of the present invention; 

Figure 8 illustrates the shape of a mask in accordance 
with a preferred embodiment of the present invention; 

Figures 9A - 9J illustrate the use of a mask in obtaining 
30 super-resolution, in accordance with a preferred embodiment of the 

present invention; 

F i g ure 1.0 il luGt r at c s an e nh a nc e d imag e obtain ed b y 



applying super-resolution to the low resolution CCD captured image 
illustrated in Figure 3B, without the use of a mask; and 



35 Figure 11 is a simplified illustration of an alternating 

("checkerboard'*) mask attached to a sensor panel of array detectors in an 
image sensing device, in accordance with a preferred embodiment of the 
present invention. 
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DETAILED DESCRIPTION OF A PREFERRED EMBODTMENT 



The present invention concerns digital images captured 
by digital image acquisition devices, such as digital cameras and 
5 scanners, and provides a method and system for obtaining super- 

resolution imagery. Super-resolution refers to the ability to obtain a 
resolution of a digital image that is greater than that native resolution of 
the image acquisition device. The present invention uses multiple low- 
resolution captures of a picture to obtain a high resolution digital image. 

10 For example, if 64 images of an object are captured by a CCD camera 

having a spatial-resolution of 32 x 32 pixels, the present invention can be 
used to integrate the captured data and provide a digital image of the 
object at a spatial resolution of 256 x 256 pixels. The present invention 
makes efficient use of captured image data, in that there is a direct one- 

15 to-one relationship between the number of images capmred and the ratio 

of resolution enhancement. That is, if K images are captured then the 
present invention enhances resolution of the capture device by a factor of 
K. 

Additionally, the present invention applies to acquisition 
20 devices, regardless of their dynamic color range. In the example 

described above, the CCD camera can have as low as 1 bit per pixel color 
depth, or as high as 8 bits per pixel and higher, and the present invention 
is applicable. 

As used in the present specification in reference to 
25 digital images, the terms "low resolution" and *'high resolution" are 

intended to be relative terms, used to indicate an advantage of the present 
invention in providing digital images of an input object at a higher 
resolution than those produced by an image acquisition device. 

Reference is now made to Figure 1, which illustrates a 
30 one-dimensional cross-section of a prior art image sensing device, such 

as a CCD camera or a scanner. An image sensing device 100 consists of 
an aaay of deteciu is 110 in a-sensin^ panel (not sh o wn), each of wh i ch 
measures a light intensity of a scene 120 at a specific pixel location. 
„Ught-j:aysJJ_0,emanatingAom_scene 120 are diffracted by a lens 140. 



35 and arrive at detectors 1 10. 

The array of detectors 110 in the sensing panel is two- 
dimensional, and the one-dimensional array of detectors illustrated in 
Figure 1 is but a single cross- section of the two-dimensional array. 
Similarly, Figure 1 iUusuates a single cross-section of scene 120. The 

40 direction of the one-dimensional array of detectors is denoted by a 
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variable A\ and the light tniensity of the scene is correspondingly denoted 
by u(x). The variable x is a continuous variable, and scene 120 
corresponds to a continuous signal. The array of detectors 110 samples 
scene 120, and produces a discrete digital image, comprising pixel 
5 locations and pbcel color intensities. The spacings between detectors 

corresponds to the pixel size for the sampled digital image, and is 
referred to as the ''pitch." 

Reference is now made to Figure 2, which illustrates a 
one-dimensional cross-section of a^ prior an pixel sensitivity 

10 (apodizaiion) function, for modeling averaged light intensity arriving at a 

detec-tor. The measured light intensity at each detector 110 (Figure 1) is 
not a measure of the Ught intensity at a single point location of scene 
120. Instead, it is an average intensity, averaged by a pixel sensitivity 
function g(x), or apodization. The pixel sensitivity function is illustrated 

15 as a one-dimensional function g(x) in Figure 2, but g(x) is only a single 

cross-section of a two-dimensional pixel sensitivity function g(x, y). 

The support of the two-dimensional pixel sensitivity 
function corresponds to the region that is sensitive to radiation. This 
region typically extends across an area that is contained within the area 

20 between neighboring pixel centers. The ratio between the area of the 

two-dimensional support of the pixel sensitivity function and the area 
between pixel centers containing it is referred to as the "fill factor." 

It is the averaging with g(x) that gives rise to 
complications in obtaining exact super-resolution in prior art methods. 

25 Reference is now made to Figures 3A - 3C, which 

illustrate the results of computer simulations of a preferred embodiment 
of the present invention, as applied to CCD cameras. Figure 3 A shows an 
input picture with spatial resolution of 256 x 256 pixels. Figure 3B 
shows an image of the input picture, captured by a CCD camera having a 

30 spatial resolution of 32 x 32 pixels, and 8 bits per pixel color depth. 

Figure 3C shows the reconstmcted image obtained by the present 
invention by u s i ng 64 pi ctu res like the one il lust rat e d in F igu re 3 B . It 



can be seen that substantially aU of the spatial details contained within 
the input picture are reconstructed in the image of Figure 3C. 

35 Reference is now made to Figures 4A - 4D, which 

further illustrate the results of computer simulations of a preferred 
embodiment of the present invention, as applied to CCD cameras. 
Figure 4A shows a low resolution image captured from the input picture 
illustrated in Figtire 3A, using a CCD camera having a spatial resolution 

40 of 32 X 32 pixels and a color depth of 1 bit per pixel. It can be seen that 
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v^^Fisure 4A is a black and while ima^^^i 



the image shown^fT Figure 4A is a black and while imagt^ i-igure 4B 
shows the reconstmcied image obtained by the present invention by 
using 64 pictures like the one illustrated in Figure 4A. It can be seen that 
although Figure 4B is not identical to Figure 3A, it nevenheless is a 
5 much bener approximation than is Figure 4A, both in terms of spatial 

resolution and in terms of dynamic color range. 

Similarly, Figure 4C shows a low resolution image 
captured from the input picture illustrated in Figure 3A, using a CCD 
camera having a spatial resolution of 32 x 32 pixels and a color depth of 
10 2 bits per pixel. It can be seen that the image shown in Figure 4C is 

comprised of four colors. Figure 4D shows the reconstructed image 
obtained by the present invention, by using 64 pictures like the one 
illustrated in Figure 4C. It can be seen that Figure 4D is almost identical 
to Figure 3A. 

15 When the present invention was applied to a CCD 

camera having a spatial resolution of 32 x 32 pixels and a color depth of 
4 bits per pixel (not shown), the reconstructed image (not shown) was 
indistinguishable from the original. 

Reference is now made to Figures 5A - 5C, which 

20 illustrates the results of computer simulations of a preferred embodiment 

of the present invention, as applied to barcode readers. Shown in Figure 
5A is an original barcode. Shown in Figure 5B is the image captured by 
a bar code reader. As a result of the limited resolving capability of the 
barcode reader, the captured image shown in Figure 5B is substantially 

25 different from the barcode shown in Figure 5A. Shown in Figure 5C is a 

reconstructed image of the barcode using the present invention, based on 
eight vertical replications. A comparison between Figure 5A and Figure 
5C indicates that the present invention successfully overcomes the 
limitations of conventional barcode readers in resolving scanned 

30 barcodes. 

The present specification describes the process for 
acq u iring m ult iple-lew-resolurien im ag es f-rom-na-siTtgle^iotu r e , and th e 
way to combine the multiple low resolution image data to generate a high 
resolution image. 

35 

Conventions and Notation 

In the ensuing description it is assumed that a sensing 
device samples an object being acquired in Unes, such as horizontal lines, 
using pixels per line. The present invention is preferably applied 
40 separately in two dimensions, such as in "a horizontal dimension and a 
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vertical dimension, lo increase the pixel resolutions in each of the 
dimensions. Since the applications of the present invention in each of 
the dimensions are similar, for the sake of clarity and definiteness the 
present specification describes a preferred embodiment of the present 
5 invention in a single dimension, such as a horizontal dimension. In 

particular, rather than use two coordinates (x, y) in the discussion below, 
a single x coordinate is used, and rather than work with double integrals 
and double sums, single integrals and single sums are used. Similarly, 
two-dimensional dpubly periodic masking of a two-dimensional array of 
10 detectors in a sensing panel is described hereinbelow using a one- 

dimensional periodic masking function m(x). 

When the sensing device samples a line, the detected 
energy of the n-th pixel is given by 

u[n] = \u{x-\-nAx)g{x)dx , n ^ 0, I N-l , (1) 

15 where u(x) denotes the captured object along a specific line, iSx denotes 

the pbcel width and g(x) denotes the pixel sensitivity (i.e. apodization). 

Multiple Image Acquisition 

In a preferred embodiment of the present invention, the 
20 sensing device samples a line K times, each time with a shift in the 

captured object relative to the capture device by an amount of A;c / AT. 
Letting ujn] denote the detected energy of the n-th pixel in the k-lh 
sampling, Equation 1 generalizes to 

«fr[^]= [uix-^nAx-^k — )g{x)dx , 
L ^ 

25 « = a 1. N'l; k = 0, 1 K-1 . (2) 

Together, the various samples ujn] provide KN samples given by 

Ax 



y[n]= juix-i-n—)g(x)dx , n = 0, A KN-l 



(3) 



In an preferred embodiment of the present invention, the 
shifts in successive image acquisitions by Ax / a: are impiementea Dy 
30 using vibrations of the sensing device. For example, the sensing device 

A an4nft-plaf^.d^Tt-a- vjhmting-platfeiTn . - ^^ 

are estimated by vibration sensors or by use of appropriate algorithms. 
The vibration parameters are used to synchronize the sampling of the 
object so that the sampling occurs at times when the vibrated sensing 
35 device is located at appropriate sub-pixel shifts. 

In an alternate embodiment of the present invention, an 
assembly of one or more rotated mirrors can be used to shift successive 
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ofa sensine device by sub-pixel shifts. The 



fields of view ofa sensing device by sub-pixel shifts. The rotated 
mirrors reflect the scene to the sensing device. 

In yet another embodiment, the shifts in successive 
image acquisitions by dec / K can be implemented using an optical 
5 element instead of the imaging lens of the sensing device. The optical 

element is anached to the aperture of the sensing device, and ser\'es ILke 
a grating to create replicated views of an object. Typically gratings 
result in replicas since the Fourier transform of a grating is an impulse 
train of dg)ta functions. This embodiment. using an optical element to 

10 create replicated views is panicularly well suited for objects that are 

sufficiently small so that the replicas do not overlap. 

The abovementioned optical element can be designed, 
for example, by a multi-facet lens. It can also be designed by an 
algorithm such as the one described in Z. Zalevsky, D. Mendlovic and A. 

15 W. Lohmann, Gerchberg-Saxton algorithm applied in the fractional 

Fourier or the Fresnel domain. Optical Letters 21, 1996, pages 842 - 
844, the contents of which are hereby incorporated by reference. 

It should be apparent to those skilled in the art that it is 
not necessary that the multiple acquired images be separated by identical 

20 sub-pixel shifts. The present invention applies to multiple acquired 

images separated by sub-pixel shifts of arbitrary sizes. Moreover, 
successive shifts between acquired images do not have to be sub-pixel. 
The shifts can be larger than one pixel shifts, as long as the multiple 
acquired images are all non-integral pixel shifts from one another. 

25 Specifically, the relative shifts of the multiple acquired images with 

reference to a fixed origin can be set to values r^x, r^z^, Tj^-.^Ax, as 
long as the differences - r. are non-integral for any distinct indices / 
and j. The case of equal sub-pixel shifts described above corresponds to 
r, = k/K. 

30 Reference is now made to Figure 6, which illustrates 

three samplings of an object by a sensing device, separated by sub-pixel 
s hifts , il l— acctJTdance— with a p itfciied e mtwdtLnc iil o f tlie— piif^i;Ttt= 
invention. The object is indicated by a curve u(x), denoted by reference 
nnmeral-6-li). The^jjrve u(x) denotes the color of the object along a line. 



35 as a function of position, x. Also indicated in Figure 6 is a curve g(x) 

denoting the sensitivity of a pixel, denoted by reference numeral 620. 
The curve g(x) is a characteristic of the sensing device, and indicates how 
color values are localized by averaging. 

Each row in Figure 6 indicates a samphng. The first row 

40 samples the object u(x) at positions indicated by dashed vertical lines 
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630. The six pixel values are denoted by uJO], ujl], ... u/57. The 
second row samples the object nix) at shifted positions indicated by 
dashed vertical lines 640. The six pixel values are denoted by uJO], 
ujl], ... uJ5]. The third row samples the object u(x) at shifted positions 
5 indicated by dashed venical lines 650. The sbc pixel values are denoted 

by tiJO], iiJI]. ... 

The samplings indicated by the second and third rows is 

often referred to as "sub-pixel" sampling, since these samplings are 

centered at fractional pixel locations. Specifically, in Figure 6, the 

10 samples in the second row are centered at the one-third pbcel locations. 

and the samples in the third row are centered at the two-thirds pixel 

locations. The value of K for the geometry illustrated in Figure 6 is = 

i. The three sets of samples together form the eighteen samples y[0] = 

UofO]: yin = y[2] = 11,10]. y[3] = uJlL .... y[17] = u/57, form 

15 the full set of samples at all of the pixel thirds locations. These eighteen 

^x ^^- ^ 

samples y[n] are used to reconstruct the values u(0), «(— ) > «(-—;:;—) » • • 

u( — - — ) of the object u(x), 
3 

The present invention uses the samples yfn] to 

Ax 

approximately reconstruct the values of u{n—). Determining these 

20 reconstructed values achieves super-resolution with a resolution 

enhancement factor of K, since the sensing device has a resolution of 
pixels per line, and the reconstructed signal has KN pixels per line. 

In order to describe the reconstmction of the values 

A V- 

u{n — ) from the samples y[n], a classical result from signal processmg 

25 is used, which relates the discrete Fourier transform of a sampled analog 

signal to the continuous Fourier transform of the analog signal. 

A Bit of History: Frequency Representation of Sampling 

_=_ In ti^e^-ensuing — description, s(r) is nse rl To de n ote . 

30 continuous time signal, and S(jQ.) is used to denote its continuous-time 

Fourier transform, defined by: 



S{jn) = £^s{t)€-^'^dt . (4) 

Periodic sampling of s(t) with a sampling period of T produces a discrete 
lime signal ^^7, defined by: 
35 s[n] = s(nT) , (5) 

The discrete-time Foiuier transform of s[n] is denoted by Sie''^), and is 
defined by: 
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S {€'")= 'Zs[n]e-''^ . (6) 

An age-old result, referred to as the Poisson Summation 
Formula and which is one of the most prominent formulas used in 
discrete signal processing, provides a relationship between the 
continuous-time Fourier transform S(jQ) of a continuous time signal and 
the discrete-time Fourier transform S(e'^) of the sampled signal. 
Specifically, the Poisson Sunamation Formula states that 

t.(nr) = lt5(-;^). (7) 

In fact, more generally the Poisson Summation Formula establishes that 



2rJct 



10 ±s{t + nT) = ^±.S{-j^)e''"' . (8) 

of which Equation 7 above is a special case corresponding to f = 0. 

A simple proof of Equation 8, as indicated on page 210 
of Rudin, V/., ''Real and Complex Analysis: Second Edition " McGraw- 
Hill, 1974, is as follows. The function /frj defined by 

15 /(r)= X^a + nD (9) 

is a periodic function with period T. As such, it has a standard Fourier 
series expansion of the form 

f{t)=±c,e''^ . (10) 



20 c,=-^lf{t)e'~dt . (11) 



where the Fourier coefficient is given by 

-f 

Upon substituting Equation 9 into Equation 11 it can be seen that 

c.=j\'_AOe'^dt = ^Si-j^), (12) 

from which Equation 8 follows when Equation 12 is substituted back 
into Equation 10. 

.car 

25 it the function s{t)e ^ is~as"eti-m Equation 8, instead of 

the function s( t), one obtains the result 



By setting f = 0 in Equation 13 one arrives at the familiar sampling 
equation 
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which appears as Equation 3.20 in A. V. Oppenheim and R. W. Schafer, 
''Discrete-Time Signal Processing," Prentice Hall, 1989. Equation 14 
indicates that the discrete Fourier transform of a sampled analog signal is 
comprised of periodically repeated copies of the continuous Fourier 
transform of the analog signal. The copies are shifted by integer 
multiples of the sampling frequency and superimposed. Equation 14 is 
often expressed in terms of delta functions; namely, that the Fourier 

transform of an impulse train ^5(r-n7) is itself an impulse train 



) 



T 

10 It is noted that Equation 14 can also be derived directly 

from Equation 7 by using s{t)e ^ instead of s(t). 

For values of co that are small enough to avoid 
significant aliasing in Equation 14, say co < , one can approximate the 

simis on the right hand sides of Equations 13 and 14 by the term with k - 
15 0, When this approximation is made, it in turn leads to the 

approximation 

%s{t-^nT)e-^''^ ^S{e^'')e' , - < • (15) 

It is noted that as the sampling period, T, decreases, the approximation in 

Equation 15 becomes more accurate. The increase in accuracy is due to 

20 the fact that the aliasing, or overlap, between the copies on the right hand 

271: 

sides of Equations 13 and 14 decreases, since the shifts — between 
copies increase. 

The present invention uses a version of Equation 15 to 
obtain super-resolution. Specifically, instead of sampling s[n] = s{nT) as 
25 in Equation 5, the sampling is done through an integral 

y[n] = [j{t ^ nT)g{:t)dt , (16) 

where ^(t) represents the sensitivity, or apodization, of a pixel. 

Multiplying both sides of Equation 15 by g(t) and integrating, results in 

CO 



r(e>")^5(g^")G(-j|-) , 



<n,, (17) 



which is the form used in the present invention. 

Referring back to Equation 3, it follows from Equation 

17, upon replacing Tby , that 
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G(-j—) 



(UK 



Ax 



<n,, (18) 



Ax 

where U(e"^) is the discrete Fourier transform of the samples u(n — ), 

K 

sought to be reconstructed; namely, 

5 The Fourier transform ¥(6^"^) is known from the samples y[n] in Equation 

3, and the Fourier transform G(- y^'^)is determined from knowledge of 

Ax 

the sensitivity function g(x). It is noted that as K increases, the 
approximation in Equation 18 becomes more accurate. 

Regarding the Fourier transform Y(e^^)y the values of 
10 y[n] set Fonh in Equations above- are only defined for indices n = 0, 1, 

.... KN-J, For indices n outside of this range, y[n] is preferably defined 
to be zero. With such a convention, the Fourier transform Yie"^) is given 
by the finite sum 

Yie''')= J^yMe-^'^ . (20) 

n=0 

15 In a preferred embodiment, the present invention uses 

Equation 18 to determine the Fourier transform U(e^'^), and then 

Ax 

reconstructs the values w(n — ), n = 0, 1, KN - 7, from U(^'^) by 

K 

using an inverse discrete Fourier transform. Preferably a Fast Fourier 

Transform (FFT), as is familiar to those skilled in the an, is used to 

.20 invert the spectral values U(e^). The reconstruction is approximate, 

since Equation 18 is itself only an approximation. Since it is necessary 

to reconstruct KN values of u(x), it is accordingly necessary to apply 

2tc 

Equation 1 8 at KN frequencies that are spaced apart one from the 

KN 

next, since the Fourier transform U(e^) is periodic with period 2k. In a 
prfifp FTpd embo diment of the present inv ention. Equation 18 is applied at 
the KN frequencies 

0)= — , k = 0, I .,.,KN~1. (21) 



When the Fourier transform U(^^) is inverted it is 
Ax 

possible that values of «(n — ) are non-zero outside of the index range n 

K 

Ax 

- 0, 7, KN - 7. As it is only the values of u{n — } within the index 

K 
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range n = 0, 1, KN - 7 that are of interest, the values outside this 
index range can be ignored. 

An apparent difficulty arises in implementing Equation 
18 if there are frequencies co as given in Equation 21, at which 

5 )=0, since this would entail division by zero at such 

A.r 

frequencies. 

The function G( -jO.) is typically a real-valued function, 
since the pixel sensitivity function g(x) is typically symmetric about x = 
0. Since a pixel is typically more sensitive to light in its central area and 

10 less sensitive in its outer regions, the pixel sensitivity function g(x) is 

typically ''bell-shaped,'* similar to Gaussian functions. In case g(x) 
happens to be a Gaussian function, its Fourier transform has no zeros, 
and there no divisions by zero are encountered in implementing Equation 
18. However, for other sensitivity functions, such a difficulty can arise. 

15 The present invention avoids such a difficulty by using 

specially constructed masks, as described hereinbelow. 

Masking 

In order to avoid division by zero in Equation 18, the 
20 present invention preferably employs an optical device to remove zeros 

of G(- j ) that are less than the frequency Cl^. The frequency Cl^ is 

Ax 

the maximal frequency in the reconstructed data. 

The present invention preferably attaches a mask in the 
form of a fine transmission grating with a period of Ax to the sensing 

25 device panel. As mentioned hereinabove, the mask under discussion is 

really a two-dimensional doubly periodic mask that covers a two- 
dimensional array of detectors. For the sake of clarity and definiteness it 
is being described as a single dimensional periodic mask, with the 
understanding that the one-dimensional analysis presented herein applies 

30 to each of the two dimensions of the sensing device panel. 

Reference is now made to Figure 7, which is a 
simplified illustration of a mask attached to a sensor panel of array 
'^ ^eteet^ rs~in~^~im'agc"^ en5inf^evieef4n^€^o rda^ with a~pr c f e rred" 
embodiment of the present invention. A sensor panel 700 in an image 

35 sensing device contains a two-dimensional array of detectors 710. 

Detectors 710 are depicted as individual cells within sensor panel 700. 
The array of detectors is periodic, with periodic spacings Ax and Ay in 
the horizontal and vertical dimensions, respectively. The spacings A.x: 
and Ay correspond to the pixel spacings in the sampled digital image 
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obtained from tnHmage sensing device. A masking pati^Pf20 in the 

form of a fine sub-pixel transmission grating is used to generate a doubly 
periodic mask, with respective periods Ax and Ay in the horizontal and 
vertical dimensions. In a preferred embodiment of the present invention, 
5 the doubly periodic mask is attached to sensor panel 700. Each detector 

710 is masked by the same masking pattern 720. 

Use of a mask decreases the effective size of each pixel 
so that the Fourier transform G(jQ) of its sensitivity function has a wider 
band, and its zeros accordingly move to higher frequencies. The 

10 apodized pixel, determined by the transmission pattern of the grating, is 

preferably designed in an iterative manner, as described hereinbelow, so 
that the bandwidth of the Fourier transform G(JCI) of its sensitivity 
function is wider than the maximal frequency to be reconstructed. Use 
of a transmission grating has the disadvantage of decreasing the energy 

15 sensed at each pixel, since it blocks out some of the illumination. 

A mask is described mathematically by a spatial function 
m(x), preferably taking values 0 and 1, The effect of a mask is to modify 
the pixel sensitivity function from g(x) to p(x) = g(x)m(x). An objective 
of choosing a suitable mask is to obtain a function p(x) whose Fourier 

20 transform does not have zeroes at frequencies that are between 0 and Q , 

so that when the Fourier transform P{-j^^) of the modified pixel 

Ax 

sensitivity fimction p(x) is used in Equation 18 instead of G(-y^^) , 

Ax 

there is no division by zero. 

A rationale for using a mask to avoid zeroes of G is that 
25 typically an effect of a mask is to add frequencies that may have not been 

present beforehand. 

To construct a mask as desired, the present invention 
preferably uses a grating in the form of a plurality of optically 
transparent slits. 

30 mix) = 1 - y rectA ® 5 (x - x, ) , (22) 
^ =§^ 



where M is the total number of slits, x^, x,, x^^ are the locations of the 
-Slits 3nd Sx, i5;.the_widthjofjhe £-th slit. The function rectQ is given by 




otherwise 

and ® denotes convolution. 
35 Reference is now made to Figure 8, which illustrates the 

shape of a mask in accordance with Equation 22. Such a mask consists 
of M slits 810, centered at positions x,, x,, x^ each having respective 
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widths of 5x.. Preferably, the value of M is a number between zero and 
K, and the values 5x are each approximately equal to the sub-pixel size 
A,v / K. As can be seen in Figure 8, the values of the mask m(x) are 0 
inside the slits and 1 outside the slits. The total width of the mask if A,\'; 
5 namely, the width of a pixel. The mask is intended to affect the shape of 

o(x) at sub-pLxel granularity. In practice, the granularity is discretized to 
i\x / K\ namely, the sub-pixel shift between successive images acquired 
by the image sensing device. For example, referring back to Figure 3C, 
the mask used in reconstructing the image shown therein is [0 1 0 1 1 0 0 
10 0], each of the eight entries in this array representing a sub-pixel size of 

one eighth of a pixel. Thus the function m(x) for the mask used in Figure 
3C is given for 0 < x < A.r by 

= L 8 8 8 8 • (24) 

l^' otherwise 

The sensitivity function of each masked pixel is given by 
15 p(x) = g(x)m(x) . (25) 

and it can be seen that the Fourier transform of p(x) is given by 
P ( y'Q ) = P ( Xi , ^2 , - . . , x^v/ ; Sx^ , Sx^ , . . . , 5x v/ ) = 

= G(yn) -X5x,G(7n) ©[^-^"^^sincC-i^Sx.)] , (26) 

sin z 



where sinc(z) denotes the function sinc(z) = 

z 

20 Preferably, to determine an optimal mask, a 

predetermined value of M is selected, and a search is made for position 
values X., Xy Xv^ and widths 5x^, 5x,, 5x^ for which the |P(7n)| is 

botmded away from zero as much as possible. Specifically, define 
a(Xi,X2,...,x^^ ;5xi,5x2,...,5x^^) = 
25 =mm|P(7Q;x^,X2,...,:r,;f ;5xj,5x2,..„5x^ )| , (27) 

where Q denotes the frequency up to which no significant aliasing 
occurs, as defined above. Then optimal values for x^, x,, x^, and 5x^, 
St., , ?>r^ nr^ pr^f^rpb^y HptprminpH hy 



(x^ , Xj , . x,^ ; 5xi , Sx^ , 5x^ ) = 
.30 = arg-maxa(Xi,X2,...,x^^;5xi,5x2,...,5x^^) , (28) 

where arg-max denotes the arguments x^, x,, x^^ and 5x^, 5x,, 5x^^ 
which maximize the function a. 

The search range over which optimal values of x^, x^ 
x^ and 5x^, 5x,, 5x,^ are sought is such that the total 5x^ + 5x, + ... + 
35 5x^ of all the widths does not exceed the pixel width zix. Additionally, 

each individual width 5x. must not be less than 2X, where a. is the 
wavelength of the incoming light - otherwise, the light will not be 
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Moreover, constmciion of a mask a^^ti 



affected by the mask. Moreover, constmciion of a mask an^ attaching it 
to the detectors may funher limit the widths 5x to exceed a minimum 
value of 

From energy considerations, it is clear that on the one 
5 hand, ii is desirable that each width 5x be as small as possible, since 5x. 

is proponional to the energy blocked by the mask. On the other hand, 
this must be traded off against the condition that |P(yX^)| be bounded 

away from zero. 

Specifically, if denotes the total light energy for a 
10 typical viewing target, when viewed without a mask, then the energy 

detected when a mask is present is given by 

M 

E^E.{\-^ ) , (29) 

Ax 

since the energy blocked by the slits of the mask is proponional to the 
total width of the siits combined. 
15 Thus for a given minimum energy E^^, required to activate the detectors, 

the widths 5x. are constrained to satisfy 

Jsx. <(l-^^)A.x . (30) 
1=1 

Condition (30), together with the condition that 5x,- > 5x^, for / = 7, .2, 
M, determine the search range for widths hx^, 5x„ 5x^^ over which 
20 a maximum of o(x^, x,, x^; 5x^, hx^ 5xJ is sought. 

When values of x^, x,, x,^ and hx^, hx^ hx^ are 
determined according to Equation 28, Equation 22 is used to generate an 
optimal mask. 

In and alternative embodiment of the present invention, 
25 rather than maximize a(x^, x,, x^^; 5x^, 5.x,, 5xJ, the values of 

X,, x^ and 5x, 5xj^ are determined by maximizing a weighted 
function q(x^, x^ x^; 5xj, hx^, hxj M^^p 5x, 5xJ, where w is 
an appropriate weight function. For example, w can put higher weight 
: o n smr^ ll^r th^ wiHthQ 8r ^ rh?^r prp.fprenre is given to masks 



30 having slits with smaller widths. 

Reference is now made to Figures 9 A. - 9J, which 
illustrate the use of a mask in obtaining super-resolution, in accordance 
with a preferred embodiment of the present invention. Figure 9A 
illustrates an input object, such as a barcode, represented as a function 

35 u(x). Figure 9B illustrates the Fourier transform of u(x). Figure 9C 

illustrates a pixel sensitivity function g(x). Figure 9D illustrates the 
Fourier transform of g{x). 
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Figure 9E illustrates conventional sampling of u(x) as a 
disital signal, without using super resolution enhancement. Figure 9F 
illustrates the Fourier transform of the digital signal of Figure 9E. 
Figure 9G iUustrates sampling of ii(x) as a digital signal with a resolution 
eight times higher than the sampling illustrated in Figure 9E. Figure 9H 
illustrates the Fourier transform of the digital signal of Figure 9G. 
Fisure 91 illustrates a mask used in accordance with a preferred 
embodiment of the present invention. 

Figure 9J Dlustrates the Fourier transform of a masked 
pixel using the mask of Figure 91. By comparing Figure 9D with Figure 
9J it can be seen that the spectrum of the masked pixel does not contain 
zeros within the region displayed. It can further be seen that an effect of 
the masking is to expand the spectrum of g(x). 

Reference is now made to Figure 10, which illustrates an 
enhanced image obtained by applying super-resolution to the low 
resolution CCD captured image illustrated in Figure 3B, without the use 
of a mask As can be seen in Fisure 10, anifacts in the form of venically 
oriented white stripes arise due to the presence of zeroes of ^^~J-^^ • 
When a mask is applied in accordance with a preferred embodiment of 
the present invention, the zeroes of Gi-j^) are removed, and the 

result obtained is the image shown in Figure 3C. It may thus be 
appreciated that the use of a mask serves to remove the artifacts indicated 
in Figure 10. 

Tmproved Approximation 

As explained hereinabove, the approximation in 
Equation 18 is based on the assumption that O) is small enough to avoid 
aliasing in Equation 14. A more accurate approximation can be made by 
taking into consideration the aliasing between adjacent copies of S in the 
right-hand side of Equation 14. However, in order to obtain the increase 
in accuracy, it is nec essaiy to iiici e as b die samp lin g . r ate 

It is appreciated by those skilled in the art that the 
— a<;su mption that aliasing_b_etween copies of 5 in Equation 14 is due to 
overlap between adjacent terms5(;|-) and S(y^^^:=^) is equivalent to 

the assumption that reduction of the sampling period from T to T/2 
removes aliasing. Moreover, the information at frequency co in the 
discrete Fourier transform of a signal sampled with sampUng period T 
corresponds with the information at both frequencies ca/2 and oy2 + tt in 
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the discrete F^^^r transform of the signal sampled sampling 
period 7/2. A precise formulation of this relationship appears in the 
above referenced Oppenheim and Schafer as Equation 3.76, which 
describes a frequency-domain relationship between the input and the 
5 output of a sampling rate compressor ("downsampling"). 

These observations above are a basis for an improved 
approximation used in accordance with a preferred embodiment of the 
present invention, as described hereinbelow. Specifically, replace the 
sampling period, T, by 772 in Equation 16: 

10 y[n] = [_j{t^n'^)g{t)dt . (31) 



Under the assumption that aliasing in Equation 14 is due to overlap 
between adjace 

S(e''^) becomes 



- CO CO ± 27l 

between adjacent terms o(y— ) and S{j — — — ), the approximation to 



S{e''') = ^SUj:) + jS(j'^-^) , 0<co<7t . (32) 

15 for positive values of CO, and 

S{e^n-~S{j^)-i-^S{j^^-^) , -Ti<co<0, (33) 
T T T T 

for negative values of co. 

Using similar analysis to that embodied in Equations 13 

- 18, it can be shown that 



20 U(€^)^~ 

0 



Ax Ax 



0<C0 <TC , (34) 

for positive values of co, and 
1 



2 



Ax Ax 



-7C <co < 0 , (35) 

25 for negative values of co. Equations 34 and 35 determine U(e^^) more 

accurately than Equation 18, since they incorporate aliasing between 

adjacent copies of 5 in the sampling representation for S(e^^). However, 

Ax 

they require sub-pixel shifts of between successively captured 

2,K 



images. This corresponds to a factor of 2K increase in data acquisition, 
whereas the resolution is only enhanced by a factor of K, 

An alternative approach to improving the approximation 
in Equation 18 by taking into consideration the aliasing between adjacent 
copies of S in the right-hand side of Equation 14, uses an alternating 
(''checkerboard") mask geometry. As mentioned hereinabove, the above 
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discussion is based on use of a periodic mask, with a period of A,v. By 
using instead a mask with a period of 2A.T, one effectively generates two 
pixel sensitivity functions, gjx) and g,(x), where gjx) applies to even- 
numbered detectors and gJx) applies to odd-numbered detectors. 
5 Specifically, o^(x) corresponds to g(x)m(x) and g^Jx) corresponds to 

g(x)m(x-\'Ax), where g(x) is the pixel sensitivity function for the image 
sensing device and m(x) is the overall mask. Unlike the above case 
where the mask has a period of Ax, when the mask has a period of 2Ax 
there are effectively two masks operating (namely, the mask itseli and 

10 the mask shifted by Ax), and consequently the even and odd-numbered 

pixels are processed differently. ' 

Reference is now made to Figure 11, which is a 
simplified illustration of an alternating (''checkerboard'') mask attached 
to a sensor panel of array detectors in an image sensing device, in 

15 accordance with a preferred embodiment of the present invention. A 

sensor panel 1 100 in an image sensing device contains a two-dimensional 
array of detectors 1110. Detectors 1110 are depicted as individual cells 
within sensor panel 1100. The array of detectors is periodic, with 
periodic spacings Ax and Ay in the horizontal and vertical dimensions, 

20 respectively. The spacings A-t and Ay correspond to the pixel spacings in 

the sampled digital image obtained from the image sensing device. 

In distinction to Figure 7, Figure 11 illustrates two 
different masking patterns 1120 and 1130 in the form of fine sub-pixel 
transmission gratings that are arranged in an alternating checkerboard 

25 pattern to generate a doubly periodic mask, with respective periods 2Ax 

and 2 Ay in the horizontal and vertical dimensions. In a preferred 
embodiment of the present invention, the doubly periodic mask is 
attached to sensor panel 1100. The detectors 1110 are alternatively 
masked by the different masking patterns 1 120 and 1130. 

30 When using such a mask, Equation 2 (which is one- 

dimensional) separates into two equations. 



«J"]= \uix + nAx + k—)g^{x)dx , 



^fQr^v-en^alues^ f-nr"Av=-£>,— /, K-h-^d (36) ^ 



wj/i]= {u{x-^nAx'^k—)g^{x)dx 



35 for odd values of n; k - 0, 1, K-L (37) 

By capturing twice as many images as pre\aously, namely, 2K images, 
each detector moves over a full period of the mask, namely, 2Ax, 
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10 



15 



20 



Consequently, t^^^equences of the form of Equation 3^Br acquired, 
namely, 

y^[n] = ]u{x + n—)gix)dx , n = 0. 1, .... KN-1 , and (38) 
K 



yj[;j] = \u{x^n—)g{x)dx , n = 0. 1. KN-1 . 
K 



(39) 



In turn, using the same analysis embodied in Equations 13-18, one 
arrives at the two conditions 

!',(.*)45(;f)G,(->^) + is(y5^)G,(-y5^^) , (40) 

45(;|iG,(wf)45u!^)C:(-;^) . (41) 

for positive values of co, 0 < (0 < tc, and 

y,(,*.,.is(;|,G,(-;f) + i5(,iii5L)G,(-y5i±2L) , ,42) 

y,„«,.i5U^,G,(-;|)4505^)a(-;i^) , (43) 

for negative values of co, -Tt; < co < 0, These equations can be used to 

CO CO ± 27l 

solve for the individual copies S{j—) and S{j — - — ), and then to 

recover Sf^^j from Equations 32 and 33. The final result obtained, 
expressed in terms of i7fO, is 



)5 ... (-7 ^) - )5 ^ (-7 ^) I 



Ax 



G, — )5 G, (-; — ) - G, {-j — )5 G^ (-; — -) 
A.X — Ax ^ ^ 



(44) 



for positive values of co, 0 < co < 7C, and 
C/(e^) = 



l'.(^^")5 .:^G,(-;^)-y.(e>")5 ,^G,(-;^) 



.(UK 



(UK, 



G,(-;^)5 2.^G,(-j— )-G,(-7— )5 3^G,(-;— ) 



Ax 



Ax 



(45) 



Ax ' Ax 

for negative values of co, -7C < co < 0. In Equations 44 and 45, the terms 



25 



b^¥G('jQ) denote the difference 

5^G(-jn) = G(-jn) - G(-j(n-'¥)) . (46) 

In' idistinetTOn^onjie^p pronchnjsing^quattons-^-^^^ 



whereby the improvement in accuracy stems from reducing the sub-pixel 

sampling shift in half, from — to — , the approach using Equations 

K IK 

42 and 43 obtains improvement in accuracy by acquiring 2K sampled 

Ax 

images, each shifted by — . Both approaches require collection of 

K 
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sampled data that is 2K as much data as in a single image captured by a 
sensing device. They differ in the types of masks used, and in the 
relative shifts between the multiple images acquired. 

When implemented with full two-dimensional 
5 processing, both approaches described hereinabove require as much 

data as in a single image captured by a sensing device, in order to 
enhance image resolution by a factor of two in each pixel dimension. 

Geometric Super Resolution vs. Diffractive Super Resolution 

10 The multiple image acquisition described above and 

illustrated in Figure 6 is geometric in nature; i.e., it is based on viewing 
the same spatial infoimation several times, such that in each view the 
object is shifted by a sub-pixel amount. It can be implemented by 
shifting positions of the sensing device (''temporal multiplexing'*) or, 

15 equivalently, by shifting fields of view (''spatial multiplexing") using an 

optical element that replicates views of the object with sub-pixel shifts 
between the replicas. The resolution of a system limited by geometry 
corresponds to the detection sampling system; i.e., the resolution of 
system sensors. 

20 In distinction, multiple image acquisition can also be 

diffractive in nature. The resolution of a system limited by diffraction 
corresponds to the finest detail that can pass through the system without 
being distorted, and is proportional to the size of the aperture in the 
optical system. Thus the resolution of the human visual system is limited 

25 by the extent to which the eyes are open. When an observer squints his 

eyes, the resolution of a scene being viewed decreases, since the size of 
the opening is decreased. 

For example, resolution enhancement can be 
implemented by an assembly of two moving rotated gratings, a first 

30 grating attached to the object being captured, and a second grating 

attached to a sensor of the sensing device, and moving in a direction 
— oppos ite to tha t uf llie f iisi giatii ig. Sutli an assembly can be used to 
enhance the resolution of a diffractive system, since it effectively 
increaseis the !size„Qf_the_aperture in the optical system. The movement of 



35 the grating attached to the object encodes its spatial information by a 

Doppler-like effect, and allows the information to be transmitted through 
a restricted aperture of an imaging lens. The decoding of the information 
is obtained from the second grating attached to the sensing device, which 
moves in a direction opposite to that or the first grating. Such an 
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assembly of iwo rotated gratings is practical for microscopic 
applications. 

However, in many imaging systems the object is distant 
from the obser\'er, and attachment of a grating to the object is 
5 impractical. When applied to scanning systems such as barcode readers, 

the present invention preferably simulates attachment of a moving 
grating to the object by illuminating the object with an illumination 
having a spatial structure of a grating. The illumination is shifted with 
time in order to simulate temporal motion of the grating. A specially 
10 designed diffractive optical element is attached to the illumination source 

so that a grating pattern appears in the object plane. The grating pattern 
is moved by phase modulating the light source, since linear phase 
modulations appear as shifts in a far field. 

15 Additional Considerations 

In reading the above description, persons skilled in the 
art will realize that there are many apparent variations that can be applied 
to the methods and systems described. For example, the integral in 
Equation 3 may be replaced by a discrete Gabor transform over u with 
20 window fimction g] namely, 

y[n] = J^g(p^)uam-p)^) . (47) 
p K K 

Similarly, one may use instead a wavelet or Mellin transform, as is well 
known to persons skilled in the an. 

Additionally, the shifts between the multiple images 

25 that are acquired by the sensing device are not required to be equal. 

Funhermore, the shifts between the multiple images are not required to 
be horizontally or vertically disposed, nor are they required to be 
disposed in a fixed direction. Wavelet transforms and Mellin transforms 
are particularly advantageous when dealing with unequal shifts between 

30 acquired images. 

Ad ditional ly, t he recuns u a c ti on algoi i llmi us ed in th e 
present invention, for generating a high resolution image from multiple 
_.lQw- res Qlution-iniages.-Jieed-not„be-se.par.able._Iha^ it need not be 



comprised of horizontal and vertical reconstructions. Rather, the 
35 reconstruction algorithm used in the present invention can be a genuine 

two-dimensional algorithm. 

In the foregoing specification, the invention has been 
described with reference to specific exemplary embodiments thereof. It 
will, however, be evident that various modifications and changes may be 



Atty. Docket No. 33,827 



-24- 




made to the specific exemplary embodiments without departing from the 
broader spirit and scope of the invention as set forth in the appended 
claims. Accordingly, the specification and drawings are to be regarded 
in an illustrative rather than a restrictive sense. 
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CLAIMS 

What is claimed is: 



1 1. A method for enhancing the resolution of an image 

2 sensing device, comprising the steps of; 

3 attaching a mask to a panel of detectors in an image 

4 sensing device; 

5 generating multiple fields of view, the multiple fields of 

6 view being related to one another by sub-pixel shifts; 

7 acquiring multiple images with the image sensing device 

8 from the multiple fields of view; and 

9 combining the multiple images into an enhanced image 

10 of higher pixel resolution than the pixel resolutions of the multiple 

1 1 images. 

1 2. The method of claim 1 wherein the image sensing 

2 device is a scanner. 

1 2, The method of claim 2 wherein the image sensing 

2 device is a barcode reader. 

1 4. The method of claim 1 wherein the image sensing 

2 device is a CCD camera. 

1 5. The method of claim 1 wherein the mask is a fine 

2 transmission grating. 

1 6. The method of claim 5 wherein the panel of detectors 

2 • comprises a periodic array of detectors, and the mask is periodic witn 

3 period equal to that of the periodic array of detectors. 

1 7, The method of claim 5 wherein the panel of detectors 

2 comprises a periodic array of detectors, and the mask is periodic with 

3 period twice that of the periodic array of detectors. 
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1 8. The method of claim 5 further comprising the step of 

2 deteimining the number of slits, the positions of the slits and the widths 

3 of the slits in the fine transmission grating. 

1 9. The method of claim 8 wherein said determining step 

2 determines the number of slits, the positions of the slits and the widths of 

3 the slits, so that the mask eliminates zeroes in a Fourier transform of a 

4 function representing a sensitivity of a pixel, within a prescribed range of 

5 frequencies. 

1 10. The method of claim 9 wherein the widths of the slits 

2 are constrained so as not to block more than a prescribed fraction of light 

3 energy from reaching the detectors. 

1 11. The method of claim 9 wherein the widths of the slits 

2 are constrained so as not to be smaUer than a prescribed minimum width. 

1 12. The method of claim 9 wherein said determining step 

2 uses a weight function specifying relative weights to be placed on values 

3 for the widths of the slits. . 

1 13. The method of claim 9 wherein the mask is of the form 

2 m(x)=l-y recti )®5{x-x.) , 

1 14. The method of claim 1 wherein said generating step uses 

2 natural vibrations of a platform supporting the sensing device. 



1 15. The method of claim 1 wherein said generating step 

2 uses at least one mirror to shift fields of view of the image sensing device 

3 by sub-pixel shifts. 



1 16. The method of claim 1 wherein said combining step is 

2 applied separately to rows and columns of pixel data. 
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1 

9 



17. The method of claim 16 wherein said combining step 

uses the approximation 



Uie"')- 



to determine the rows and columns of pixel data of the enhanced image. 



18. The metho'd of claim 17 wherein said combining step 

comprises applying an inverse Fourier transform to values of U(^'^). 



1 

9 



4 

5 



7 
8 
9 



19. The method of claim 16 wherein said combining step 

uses the approximation 
1 



CO 

ru'^)/G(-y^) + y(/ ' )/G{-j 

Ax Ax 



for positive values of o), and 



0+2- 



0 <co <Tr 
.((0+27U)/: 



Yie^')/Gii-j^)^Yie ' )IG{--] 

Ax Ax 



-7t <C0 < 0 , 

for negative values of co, to determine the rows and columns of pixel data 
of the enhanced image. 



1 

9 



20. The method of claim 19 wherein said combining step 

comprises applying an inverse Fourier transform to values of i7fO- 



1 

0 



21. The method of claim 16 wherein said combining step 

uses the approximation 



3 U{e^) = 


{e ^ )5 G, (-;■ ^) - K, )5 G, (-; ^) 


1^ 









c,(-y!g^)5„G,(-y^)-G,(-;!g^)5^G.(-;g^) 



6 



"ST" 



for positive values of o), 0 < co < 7C, and 



Y, {e^ )5_^G, (-7 ^) - y. (e^ )5_^G, (- j ^) 



G,(-j^)b G. (-; — ) - G, (-; — )5 ^G, (-7—) 



. Ax 
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8 for negative values of co, -re < co < 0, and wherein the terms Sv^Gf-^Qj 

9 denote the difference 

1 0 b^c('jn) = G(-jQ) - Cf-jin-'V)) , 

1 1 to determine the rows and columns of pixel data of the enhanced image. 

1 22. The method of claim 21 wherein said combining step 



comprises applying an inverse Fourier transform to values of Uie'"^), 



1 23. The method of claim 1 wherein said combining step uses 

2 a Gabor transform. 

1 24. The method of claim 1 wherein said combining step uses 

2 a wavelet transform. 

1 25. The method of claim 1 wherein said combining step uses 

2 a MeRin transform. 

1 26. A system for enhancing the resolution of an image 

2 sensing device, comprising: 

3 an image sensing device comprising a panel of 

4 detectors; 

5 a mask attached to said panel of detectors; 

6 a motion generator generating multiple fields of view, 

7 the multiple fields of view being related to one another by sub-pixel 

8 shifts; 

9 image acquisition circuitry housed within said image 

10 sensing device acquiring multiple images from the multiple fields of 

1 1 view; and 

12 a combiner combining the multiple images into an 

1 3 e nliaiic c d i mag e of li i gher pix el r eselutjen-th an the pix e l r es ol utions of 

14 the multiple images. 



1 27. The system of claim 26 wherein said image sensing 

2 device is a scanner. 

1 28. The system of claim 27 wherein said image sensing 

2 device is a barcode reader. 



Atty. Docket No. 33,827 



-29- 



1 29. The sysiem of claim 26 wherein said image sensing 

2 device is a CCD camera. 

1 30. The sysiem of claim 26 wherein said mask is a fine 

2 transmission grating. 

1 31. ...o The system of claim 30 wherein said panel of detectors 

2 comprises a periodic array of detectors, and said mask is periodic with 

3 period equal to that of said periodic array of detectors. 

1 32. The system of claim 30 wherein said panel of detectors 

2 comprises a periodic array of detectors, and said mask is periodic with 

3 period twice that of said periodic array of detectors. 

1 33. The system of claim 30 further comprising a grating 

2 generator determining the number of slits, the positions of the slits and 

3 the widths of the slits in the fine transmission grating. 

1 34. The system of claim 33 wherein said grating generator 

2 determines the number of slits, the positions of the slits and the widths of 

3 the slits, so that said mask eliminates zeroes in a Fourier transform of a 

4 function representing a sensitivity of a pixel. 

1 35. The system of claim 34 wherein the widths of the slits 

2 are constrained so as not to block more than a prescribed fraction of light 

3 energy from reaching the detectors. 

4 26^ , The s y stem of clHim 34 w h e rein the -wi dth s o f the, slit s 

2 are constrained so as not to be smaller than a prescribed minimum width. 



1 37. The system of claim 34 wherein said grating generator 

2 uses a weight function specifying relative weights to be placed on values 

3 for the widths of the slits. 

1 38. The system of claim 34, wherein said mask is of the 

2 form 
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A/ 

w( jc) = 1 - ^ recti:^) ® 5 (.v - x.) . 

i=\ O.T- 



1 39. The system of claim 26 wherein said motion generator 

2 uses natural vibrations of a platform supponing the sensing device. 

1 40. The system of claim 26 wherein said motion generator 

2 ^ uses at least one mirror to shift fields of view of the image sensing device 

3 by sub-pixel shifts. 



1 41. The system of claim 26 wherein said combiner is 

2 appUed separately to rows and columns of pbcel data. 

1 42. The system of claim 41 wherein said combiner uses the 

2 approximation 

3 



G{'J—) 
Ax 

to determine the rows and columns of pixel data of the enhanced image. 



1 43. The system of claim 42 wherein said combiner applies 

2 an inverse Fourier transform to values of Ui^"^). 

1 44. The system of claim 41 wherein said combiner uses the 

2 approximation 

1 



0 



Ax Ax 



4 0 < CO < Tt , 

5 for positive values of CO, and 



9 



Ax Ax 



- Ji < o) < 0 , 

8 for negative values of CO, to determine the rows and columns of pLxel data 

9 of the enhanced image. 



1 45. The system of claim 44 wherein said combiner 

2 comprises applying an inverse Fourier transform to values of U(e^). 
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6 



S 
9 

10 
11 



46. The system of claim 41 wherein said combiner uses the 

approximation 



^-S-J — )S 2.-jr — ) - {-J — )5 G, (-7 — ) 



for positive values of co, 0 < co < tc, and 



-— A-v -— Ax 



)S 2.i^Gj(-y— )-Gi(-;— )5 ,^^G,(-7— ) 
A,v Am Ax — — Ax 

Ax At _ 

for negative values of co, -tc < (O < 0, and wherein the terms b^vC(-jQ) 
denote the difference 

b^G(-jn) = G(-jn) - G(^j(n-'¥)) , 

to determine the rows and columns of pixel data of the enhanced image. 



1 

9 



47. 



The system of claim 46 wherein said combiner 



comprises applying an inverse Fourier transform to values of U(e^). 



1 

9 



48. The system of claim 26 wherein said combiner uses a 

Gabor transform. 



49. The system of claim 26 wherein said combiner uses a 

wavelet transform. 



1 
2 



50. The system of claim 26 wherein said combiner uses a 

Mellin transform. 



1 51. A method for enhancing the resolution of an image 

2 ,sensingjieAlc£^cAmprisinfijhe_ste.psj3fc 

3 creating replicas of fields of view using an optical 

4 element attached to an image sensing device; 

5 acquiring multiple images with the sensing device from 

6 the replicas of fields of view; and 
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7 combining the multiple images into an enhanced image 

8 of higher pixel resolution than the pixel resolutions of the multiple 

9 images. 

1 52. The method of claim 51 wherein the optical element is a 

2 diffractive optical element. 

1 53. The method of claim 51 wherein the optical element is a 

2 reflective optical element. 

1 54. The method of claim 51 wherein the optical element is a 

2 combined diffractive and reflective optical element. 

1 55. The method of claim 51 wherein the optical element 

2 produces an effect of a grating. 

1 56. The method of claim 51 wherein the optical element is 

2 multi-faceted. 

1 57. The method of claim 51 wherein the replicas of fields of 

2 views are non-overlapping. 

1 58. A system for enhancing the resolution of an image 

2 sensing device, comprising: 

3 an image sensing device; 

4 an optical element attached to said image sensing 

5 device, said optical element being such as to create replicas of fields of 

6 view; 

7 image acquisition circuitry housed within said image 

8 sensing device acquiring midtiple images from the replicas of fields of 

9 view; and 

10 a~ct>Tril5tn'eT~C'omt)inxn^~ttie multiple uiiages uito an 

1 1 enhanced image of higher pixel resolution than the pixel resolutions of 

12 the midtiple images. 

1 59. The system of claim 58 wherein said optical element is a 

2 diffractive optical element. 
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1 60. The system of claim 58 wherein said optical element is a 

2 reflective optical element. 

1 61 . The system of claim 58 wherein said optical element is a 

2 combined diffractive and reflective optical element. 

1 62. The system of claim 58 wherein said optical element 

2 produces an effect of a grating. 

1 63. The system of claim 58 wherein said optical element is 

2 multi-faceted. 

1 64. The system of claim 58 wherein the replicas of fields of 

2 view are non-overlapping. 



^ For the Applican t, 

Sanford T. Colb & Co. 
C: 36315 
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